ON-LINE BENCHMARKING SYSTEM AND METHOD 



R ^KHRQUND OF THE INVENTION 
F-jfild of the Invention 

The present invention is related to the field of 
corporate performance evaluation and improvement and, more 
particularly, to an automated on-line system and method for 
providing a comparative, statistics-based analysis of industry peer 
performance through benchmarking. 

Description of the^ Related Art 

Benchmarking is an essential element of corporate 
performance evaluation worldwide. A "benchmark" is a standard of 
performance or a point of reference that serves as the basis of 
understanding relative standing or performance. Benchmarking 
involves comparing the vital statistics of one enterprise against 
those of a peer group of similar companies. Whether related to 
personnel, capital investment, customer care, marketing or 
administrative support operations, when decisions are to be made 
managers require as much information as can reasonably be obtained. 
Benchmarking is critical to identifying those aspects of corporate 
performance requiring improvement and formulating a knowledge -based 
plan of action to achieve objectives. 




Benchmarking, in some form, has always existed. More 
recently, however, there has been a major push in many industries 
to identify the metrics that are central to success, and to find 
ways to accomplish more effective benchmarking. Some companies 
have banded together to exchange data among themselves on a formal 
or informal basis. However, often the* data is partial in scope and 
limited in geographical reach. Similarly, some consulting 
companies have developed benchmarking practice units, but these are 
only able to provide "snapshot" studies to clients and are often 
accomplished pursuant to engagements that are simply cost 
prohibitive for many industry players. 

Customer Relationship Management (CRM) operations, among 
them call centers, lend themselves particularly well to 
benchmarking because so many measurable elements of these centers 
are collected with total objectivity through the use of technology, 
A need exists among such operations for cost-effective benchmarking 
services that provide valuable comparison data with both peer 
groups that have been specifically selected as well as broader peer 
groups worldwide. 

fiTTMMARY OF THE INVENTION 

In view of the foregoing, one object of the present 
invention is to provide an on-line benchmarking service accessing 
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a comprehensive database of industry performance metrics. 

Another object of the invention is to provide industry 
professionals with a tool for evaluating company performance and 
targeting areas for improvement through an automated system 
accessible via a distributed computer network such as the Internet. 

A further object of the invention is to enable business 
process managers to compare their performance data with that of a 
group of industry peers having characteristics specifically defined 
by the manager requesting the comparison and analysis. 

A still further object of the invention is to provide a 
benchmarking tool capable of generating a variety of data analysis 
reports which may be customized to meet the particular needs of the 
requestor . 

Another object of the invention is to provide business 
managers with a means of obtaining current and up-to-date 
comparison data in any number of dynamically changing industries in 
order to stay abreast of both strengths and weaknesses as compared 
with industry peers. 

Yet another object of the invention is to provide 
targeted benchmarking data at a very reasonable cost. 

In accordance with this and other objects, the present 
invention is directed to an on-line benchmarking system and method. 
The benchmarking model is community-based, springing from academic 
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origins while providing a wealth of additional management decision- 
oriented features to maximize its usefulness and impact. The 
system is accessible through a web site on a distributed computer 
network such as the Internet. Users enter the web site and input 
a member code to access the benchmark study datamart . In response 
to a detailed questionnaire, users submit data summarizing their 
performance metrics on-line. Input data is incorporated into the 
database. 

As a safeguard against errors in data entry, the datamart 
system includes a delinquent data checker for reviewing input data 
in order to identify data values that are suspected of having been 
entered in error. Such suspicion may arise from the data values 
themselves, in light of the bulk of data values previously entered 
for a particular index or generally anticipated values determined 
from the nature of the index. Input data is compared with 
predetermined valid data values or ranges for each index and data 
that is found to fall outside the predetermined valid data value or 
range for a particular index is placed on a list for subsequent 
verification. 

The benchmark study datamart processes the data in 
accordance with user- specif ied criteria defining a peer group and 
affords the user with a range of reporting options highlighting 
strengths and weaknesses as compared to industry peers, and 
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providing realistic solution options for minimizing performance and 
financial gaps. The datamart is thus a continually updated and 
expanded model, which involves from hundreds to thousands of 
participants. Through such broad-based participation and automated 
access approach, the results obtained through the present invention 
not only offer superior quality and breadth of data, but also 
reduced cost. 

These and other objects of the invention, as well as many 
of the intended advantages thereof, will become more readily 
apparent when reference is made to the following description taken 
in conjunction with the accompanying table and figures. 

RRTFIF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a block diagram of the on-line 
benchmarking system in accordance with the present invention; 

Figure 2 is a flow diagram of the data entry process 
using the data entry section of Figure 1; 

Figure 3 is a block diagram of the automated data entry 
system in accordance with the present invention; 

Figure 4 is a flow diagram of the overall data analysis 
process using the data analysis section of Figure 1; 

Figure 5 is a flow diagram of the operation of the 
delinquent data checker of Figure 4; 
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Figure 6 is a flow diagram of the operation of the gap 
killer of Figure 4; 

Figure 7 is a flow diagram of the report generation and 
delivery processes using the report generation/delivery section of 
Figure 1; 

Figure 8 is a sample executive report generated in 
accordance with the present invention; 

Figure 9 is a sample peer group comparison report 
generated in accordance with the present invention; 

Figure 10 is a sample peer group ranking report generated 
in accordance with the present invention; 

Figure 11 is a sample peer group performance matrix 
report generated in accordance with the present invention; 

Figure 12 is a sample performance gap analysis report 
generated in accordance with the present invention; 

Figure 13 is a sample financial gap analysis report 
generated in accordance with the present invention; and 

Figure 14 is a sample gap versus solution optimizer index 
report generated in accordance with the present invention. 



DFITAILED DESCRIPTION OF TMK PREFERRED EMBODIMENTS 

In describing a preferred embodiment of the invention, 
specific terminology will be resorted to for the sake of clarity. 
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However, the invention is not intended to be limited to the 
specific terms so selected, and it is to be understood that each 
specific term includes all technical equivalents which operate in 
a similar manner to accomplish a similar purpose. 

Benchmarking involves comparing the vital statistics of 
one enterprise against those of a peer group of similar companies. 
A few examples of the types of measurement that might be 
benchmarked, for the sake of illustration, include costs of hiring, 
training and compensating employees in various geographic 
locations; cost to handle an inbound telephone call; percent 
overhead cost structure; sales per employee; years of customer 
retention, etc. 

Virtually every type of business may benefit from 
benchmarking. For the purposes of this document, many of the 
examples used will be directed to Customer Relationship Management 
(CRM) call centers, but in no way is this intended to suggest that 
the present invention is limited to use with call centers. 

As illustratively shown in Figure 1, the present 
invention is directed to a, web- integrated on-line benchmarking 
system referred to herein as the "datamart" . The datamart 10 
includes a data entry section 12, a data analysis section 14, and 
a report generation/delivery section 16. The datamart interacts 
with users 24 over a distributed computer network such as the 
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Internet 22. 

The data entry section 12 includes a host site 18 having 
a search engine for accessing an on-line database 20. Data to be 
entered into the on-line database 20 is input through the data 
entry section 12 . The host site 18 is preferably embodied as a web 
site accessible over the Internet 22. Users 24 access the host web 
site 18 through the Internet 22, using a web enabled computer or 
other web enable device. As used herein, a web enabled computer 
shall be understood to mean any computer or other electronic device 
having the necessary software and telecommunications capability to 
connect to the Internet or similar distributed computer network for 
communications exchange . 

As an Internet web site, the host site 18 may be 
configured in a number of ways, as would be known by persons of 
ordinary skill in the art. Benchmarking data is stored in the on- 
line database 20, which may be embodied as one or more databases. 

The data analysis section 14 is connected to the data 
entry section 12 and includes a mirror database 30, a delinquent 
data checker (DDC) 32, and a gap killer 34. The mirror database 30 
is a copy of the on-line database 20. During data analysis, the 
data in the on-line database 20 remains intact while the data 
stored in the mirror database 3 0 is manipulated. Each of these 
elements within the data analysis section 14 will be discussed in 
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greater detail hereinafter. 

The report generation/delivery section 16 is connected to 
the data analysis section 14 and includes a report generator 40, a 
peer group generator/widener 42, and a report delivery device 44. 
Each of these elements within the report generation/delivery 
.section 14 will be discussed in greater detail hereinafter. 

A user wishing to initiate a benchmarking study with the 
datamart 10 accesses the host site 18 over the Internet 22. Upon 
logging onto the host site 18, the user is greeted with a "Home" 
page. As is known in the art, the "Home" page is the gateway to 
the site and provides the user with an overview of the site's 
available information and procedures. Other pages or folders may 
be selected by clicking on the visible tabs and/or listed links as 
would be known by one of skill in the art. Each page, once 
accessed, contains multiple links to other pages, as is known in 
the art. The benchmarking section is accessible through an 
appropriately named link. 

Figure 2 presents a flowchart of the process of data 
entry in accordance with the present invention, which begins by 
logging onto the host site, step 100. If the web site is being 
accessed for the first time, step 102, the user typically chooses 
at that time, step 103, to register with the site administrator, 
step 104. Registration may be accomplished following a screen- 



# 

prompted menu provided at the host web site 18. As part of the 
registration process, the user receives a password or other 
assigned security code. 

Those users whose access is not a first time access, 
steps 102, e.g., registered users, enter their individually 
assigned security code, step 106, and are granted access to the 
benchmarking section, step 108. Once within the benchmarking 
section, the user may complete an on-line survey, step 110, and the 
entered data is input automatically to the on-line database 20. 

In some instances it may be preferable to allow the user 
to choose not to register initially, step 103, but rather to enter 
survey data prior to completing the registration process; the 
present invention supports this sequence of entry as well. In this 
instance, the user may be given a guest security code or the code 
requirement may be waived or otherwise satisfied. The user then 
accesses the benchmarking section, step 108, and enters pertinent 
data, step 110. 

As shown in Figure 3, data entry may also be completed 
automatically through the use of automated telecommunications links 
such that manual input through the web site is not necessary. Such 
automatic entry is possible within certain industries, such as the 
call center industry, using existing data monitoring and generation 
equipment of the user. 
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In the example of a call center user 240, a computer 
system 242 typically monitors call handling and related functions, 
directing calls to particular operators 246. The computer system 
242, which includes a switch 244, monitors call related performance 
parameters, and generates a range of pertinent data on an automatic 
basis. Such a computer system may include an automatic call 
distributor. A variety of switches are commercially available and 
may be embodied in hardware and/or software to perform data 
monitoring and reporting functions, as would be known by persons of 
skill in the art. Such switches may be programmed or otherwise 
configured to send the generated data to the datamart 10 via an 
email over the Internet 22 at predetermined intervals, at 
particular times, etc. Using custom modules 248 developed 
specifically for interfacing with particular switches 244, or types 
of switches, the datamart 10 extracts the data needed for a 
particular report or reports from the email. In a preferred 
embodiment, the modules are embodied in software and programmed to 
gather the incoming email, locate and process the data, and 
correlate the data with the source of the email for report 
generation . 

Using the system summarized in Figure 3, data generated 
by a call center user 24 0 as a matter of regular operation may be 
electronically conveyed to the datamart 10 with no additional 



effort required on the part of the call center other than initial 
configuration of the switches. Switch coding directs the 
generation of a particular report, or reports, which is generated 
by the datamart and subsequently sent to the requesting call 
center. Through this automated data entry approach, regular 
performance evaluations may be conducted as a matter of routine 
business practice with little or no added effort on the part of the 
call center. Other industries which use automated computer 
monitoring equipment such as described in connection with the call 
centers may also be supported with automatic data entry in a 
similar manner. 

Data may also be entered manually through the web site. 
The on-line survey includes a range of questions pertinent to the 
user's particular industry. In a preferred embodiment, the on-line 
survey includes a linear questionnaire constructed for on-screen 
presentation in three frames. One frame, called the navigation 
bar, displays the section of the questionnaire currently being 
completed. A second frame, the question number frame, identifies 
question numbers completed as well as the current question number. 
A third frame displays the questions. Other presentation formats 
are of course possible and are included within the scope of the 
invention. 

Initially the user enters the questionnaire from the 
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beginning. If the user must exit the questionnaire prior to 
completion, when he or she returns to complete the questionnaire 
the last question answered will be displayed. In the preferred 
embodiment the user is not permitted to skip questions without 
providing an answer. However, zero may be accepted as a valid 
answer to most numeric questions. By accepting the widest range of 
data during the data input phase, and then filtering and removing 
inconsistent data during the data analysis phase, the data input 
phase is streamlined, minimizing the time necessary for the user to 
complete data entry. 

As questions are answered, the completed question numbers 
appear in the question number frame. The user may return to 
previously answered questions by clicking on the appropriate 
question number in the question number frame. When the user 
completes the last question in the section, the next section is 
shown automatically and the navigation bar is updated. The user 
may return to a previous section by clicking on the appropriate 
section number in the navigation bar. Completed sections may be 
displayed in one color while sections not yet completed are 
displayed in another color. 

Because the questionnaire is linear, questions having a 
prerequisite will be skipped when the prerequisite is not 
satisfied. In other words, questions having a prerequisite will 
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only be shown when the prerequisite question has been answered 
correctly. 

The data entry section also eliminates certain questions 
that do not apply to the particular user. For example, if the user 
represents a call center which does not handle any outbound calls, 
the data input section skips all questions that apply to outbound 
calls. This is done automatically by the host site through the use 
of the answers to specific questions to decide whether succeeding 
questions should be displayed. 

The data that is collected may be divided into three 
types, namely real, nominal, and text. Real values are numeric 
values and may include decimals. Nominal values are results of 
multiple choice questions, e.g., true/false, daily/weekly/monthly, 
etc. Text values are responses to open-ended questions to the user 
(e.g., what vendor do you use for reporting?) . 

As the data is collected a low level of validity checking 
is performed on the real data types by the data entry section. For 
certain questions, the entered value is compared to a range of 
realistic answers. If the data input by the user is outside the 
range, a warning may be displayed. However, the data is still 
accepted and the user may decide to ignore the warning and 
continue . 

While on-line data entry or automated electronic data 
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transfer represents the preferred embodiments, users may also 
complete hard copy questionnaires which may be mailed or sent via 
facsimile to the datamart administrator for input to the database. 

For the purposes of illustration, the contents of a 
sample survey will be summarized herein as presented for users 
involved in call centers. This summary is not inclusive of 
everything that may or could be asked but is provided to give some 
idea of the nature and scope of the information collected during 
the data entry phase. 

In the context of call centers, users may be asked 
questions relating to their specific industry area, total revenue, 
types (inbound, outbound, internal) and number of calls handled, 
and functions associated with each type of call. For inbound calls 
functions may include call routing, complaint resolution, customer 
service, dispatching, order taking, reservations, technical support 
to outside customers, etc. For outbound calls functions may 
include telemarketing and lead generation, collections, customer 
surveys, market research, etc. The call center requesting user is 
also asked to provide a breakout of costs for such things as human 
resources, computer equipment, real estate, toll-free service, etc. 

The survey includes a broad range of inbound performance 
measures including averages for speed of answer, talk time, after 
call work time, number of abandoned calls, time before abandoning, 
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time in queue, sale value and number of inbound calls per telephone 
operator shift. Performance measures also include percentage of 
calls closed on first call, percentage calls blocked, percentage of 
time spent by operators or telephone service representatives 
handling calls, adherence to schedule, and cost per call. 

Outbound performance measures include information on cost 
per call and cost per sale, sales per hour, average sale value, 
percentage of outbound calls that result in a sale, contacts per 
hour, and averages for sales revenue per operator per year, revenue 
collected per seat per shift, and sales revenue per seat per shift. 

The survey may also be used to collect information 
relating to caller satisfaction measurement, e.g., how satisfaction 
is measured, the degree of satisfaction reported, etc. The 
requesting user's data may also incorporate input, received through 
automated telecommunications links, from an independent entity 
measuring customer satisfaction. 

The survey can also collect data relating to call center 
strategy, e.g., whether focus is on acquisition or retention, basis 
for customer routing to particular agents, etc.; human resource 
management, e.g., educational level of operators, turnover, 
compensation, hiring criteria and practice, performance measurement 
criteria, training, etc.; process and knowledge, e.g., customer and 
file status information available to operators, recipients of data 
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warehousing and mining data, customer access channels, basis for 
call routing, etc.; technology solutions, e.g., hardware and 
software currently used or being contemplated for use; outsourcing, 
e.g., percentage of calls outsourced, reasons for outsourcing, cost 
of outsourcing, etc.; and facilities and design, e.g., number of 
seats, number of seats added in past year, average size of work 
space, desktop workstation equipment, etc. 

Again, this survey data summary is representative only 
and additional or alternative information may be requested 
depending on the user and the nature of the pertinent industry. 

As part of the data entry phase, after data entry, or at 
virtually any time desired by the user, the user can specify 
particular characteristics defining the peer group or a new peer 
group against which that user wishes to be benchmarked. A "peer 
group" does not necessarily connote competitors, but most often 
includes those companies having the same general profile of 
activities as the accessing user. For instance, using the example 
of call centers, a peer group might be all call centers handling 
mostly inbound calls that are mostly business-to-business in a call 
center of over 100 agents for a company with annual revenues of 
over 1 billion dollars. As another example, a peer group may be 
defined as only call centers that are inbound business- to-business, 
with between 100 and 2 00 agents, that handle at least 50% order 
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taking and order tracking calls, and that handle at least 25% 
technical support calls. Peer group parameters may be derived from 
identification parameters contained within the data input by the 
accessing user. Alternatively, the user may specifically identify 
5 peer group identification parameters that may be of a different 
scope (e.g., broader or narrower) than those actually defining its 
own business position and request comparison with this customized 
group. This ability to specifically identify the characteristics 
of a desired peer group and to be compared in a detailed manner 

10 against that group provides requesting users with a tremendous 
means of quantifying performance and identifying success strategies 
not before available in the prior art. 

The next phase is the data analysis phase, which is 
summarized in the flowchart of Figure 4. During a specified time 

15 and on a regular basis, the on-line database is copied to the 
mirror database, step 120. The remainder of the steps shown in 
Figure 4 are then performed using the data in the mirror database 
30. In a preferred embodiment, the on-line database 2 0 is copied 
to the mirror database 30 on a daily basis, such as just before 

2 0 . midnight. 

Acting on the mirror database, the delinquent data 
checker (DDC) is run, step 122, to verify the accuracy of the input 
data in an automated manner. The gap killer is also run, step 123. 
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The process undertaken by the DDC is summarized in Figure 5. The 
process undertaken by the gap killer is summarized in Figure 6. 

As shown in Figure 5, the DDC scans each completed record 
of the database, step 2 00, and compares every real value in the 
5 database to a predetermined data comparator value or limit, step 
202. In the preferred embodiment the comparison is generally made 
against an upper and lower limit. If the value being compared is 
outside the specified range, step 204, then that value is deleted, 
step 206. In addition, certain results are calculated using values 

10 in existing fields, step 208, to ensure that the ratio between the 
data points is within a predefined tolerance. For example, in the 
case of a call center, the indicated value for cost allocated to 
call operation should, when divided by the number of calls 
received, be roughly equal to an indicated cost per call. If the 

15 ratio is not within the predefined tolerance, step 210, the values 
are deleted, step 212. If the ratio is within the tolerance, step 
210, the method determines there is no delinquent data, step 124, 
and proceeds with the report generation stage, step 360. During 
the running of the DDC, the original data in the on-line database 

2 0 2 0 is untouched. 

When a value is deleted, then the record that contained 
the data is viewed as delinquent data, step 124, and is added to a 
list of data delinquents 126. Returning to Figure 4, once the 
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entire database has been scanned by the DDC, the datamart system 
automatically generates customized email messages to the users 
placed on the data delinquents list requesting data verification, 
step 128. Each such email describes the out of range value (s) and 
requests that the user confirm the validity of each piece of data 
in question. The user is provided with a convenient means of 
responding, preferably electronically such as via email, and can 
either verify that the data is indeed correct or acknowledge that 
an error was made in data entry and provide a replacement value for 
the delinquent data. In the event the delinquent data was indeed 
correct, step 130, the data is examined to determine if it is 
typical. If the data is typical 131, the upper or lower limit of 
the valid data range is changed, step 132, to specify a range that 
includes the user's data value. If the data is not typical 131, 
then the valid data range is not changed, step 133. In non-typical 
data situations, data correctness does not initiate a change in 
valid data range due to what may be considered an anomaly in the 
operation of the requesting user. For example, call centers 
relying on volunteer operators may have data values that no not 
fall within ranges which would be expected for call centers having 
paid operators . 

If the delinquent data was not correct, step 130, then 
the data is corrected, step 134, in accordance with the user's 



input . 

Removing delinquent data leaves holes, or gaps, in the 
database. In addition, when entering survey information responses 
to various indices, the user may intentionally or inadvertently 
5 . fail to provide answers for certain of the questions. These 
unanswered questions also leave gaps in the body of data. While 
entry of data to some indices is mandatory, entry of a response to 
others may be omitted without compromising the value of the survey 
as a whole. Yet the presence of gaps in the data may be interpreted 

10 by the system as indicating that the entire body of data is 
defective, aborting further processing of that survey. The gap 
killer serves to preserve useful data that might otherwise be 
rejected due to non-critical data gaps in the body of data. 

Operation of the gap killer is summarized in Figure 6. 

15 In order to neutralize the effect of non-essential gaps, the gap 
killer identifies an index containing a gap, step 320, and then 
determines an average database value for that index. The gap 
killer performs this function in two phases. First, the gap killer 
program selects two columns from the database, step 330; in a 

2 0 preferred embodiment, one of these two columns is taken from the 
beginning of the questionnaire and the other is taken from the end. 
These columns, or values for a selected parameter however 
presented, are known as the "plug- in columns" and are considered 
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essential data. 

The entire database is then scanned, step 332, to 
identify records in which both plug* in columns have been answered. 
If both plug-in columns have not been answered, step 334, those 
5 records are deleted, step 336. 

A parallel step is to analyze the remaining records and 
complete the missing data. For each record having gaps, i.e., the 
current record, a set of records known as a "plug- in group" is 
determined, step 338. The plug-in group is comprised of those 

10 users whose "plug-in column" values are within a certain number of 
standard deviations from those of the current record. Using the 
plug- in group, the gaps in the current record are filled as 
follows. If the gap in the current record is a real value, step 
340, then the gap is filled using the average value for that index 

15 from the plug-in group, step 342. If the gap in the current record 
is a nominal value, step 344, then the gap is filled with the most 
likely response, step 346. The most likely response is determined 
by counting the number of occurrences of each response within the 
"plug-in group", calculating the probability of each response, and 

2 0 then choosing a value based on these probabilities. Finally, gaps 
that would otherwise contain text values are not filled in and the 
text value is left blank, step 348. If there are additional 
records with gaps, step 350, the process loops back to step 338 to 
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determine a plug- in group for the next current record. When all 
records with gaps have been processed, the data is forwarded to the 
report generation section, step 360. 

In summary, by deleting incurable records and temporarily 
filling the gaps for non-essential indices in the remaining 
records, the gap killer completes the remaining records without 
changing the overall trends and averages. While the action of the 
gap killer may be said ' to reduce the absolute accuracy of the 
survey, this is not statistically significant. More importantly, 
by filling in certain gaps, survey information that might otherwise 
by rejected is accepted and preserved, enabling that body of survey 
data to be forwarded for further analysis. 

The third phase of the data acquisition and processing 
procedure of the present invention is report generation and 
delivery. In a preferred embodiment, this phase is completed using 
the report generator 40 and the report delivery device 44. The 
peer group generator/widener 42 establishes the parameters of the 
peer group against which the current user will be compared. As 
discussed earlier, this peer group may be determined by the 
datamart on the basis of the self -describing identification 
information obtained from the user, or the user may specify desired 
criteria or peer group identification parameters defining an 
alternative, customized peer group. 



The steps undertaken during report generation and 
delivery in accordance with the present invention are set forth in 
Figure 7. A user requests a report through the host site, step 
400, and in response to this request certain flags are set in the 
5 on-line database, including a "report request" flag, step 402. 
When the on-line database 2 0 is copied to the mirror database 30, 
the set flags are conveyed to the mirror database. 

The peer group generator establishes a peer group for the 
requesting user, step 404. A different peer group may be 
10 established every day, or for every cycle of the function of the 
present invention whether daily, or more or less frequently. 

The datamart has a variety of report formats and types 
available. For each available report, the datamart system includes 
a template containing several pages of tables and/or graphs. In 
15 response to a report request, the appropriate template is selected, 
step 406, and each requested report is generated by creating the 
appropriate tables and graphs based on the data in the "gap killed" 
database, step 408. In the preferred embodiment, every report that 
is generated is assigned a unique report identification number or 
2 0 code. Multiple copies of the same report, which may be sent to 
multiple locations, each receive a different number to facilitate 
tracking capability as well as report integrity. 

Once a set of reports is ready, the set is moved to an 
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"out box" awaiting approval, step 410. In a preferred embodiment, 
each report is converted into an Adobe Acrobat® PDF file before 
being placed in the "out box". Other file formats may, of course, 
be used. If the user had requested that the report be printed and 
5 sent through the mail then the report is moved into a 'Sprint box" 
instead of the "out box" . 

Associated with each report, or set of reports as 
appropriate, is a "ready-to-go" flag and an "OK-to-go" flag. The 
"ready- to-go" flag is set once the report has been moved to the 

10 "out box" or the "print box", step 412. Setting of the "ready-to- 
go" flag resets the user's "report request" flag. 

The reports having a "ready-to-go" flag set are reviewed 
for accuracy, and a method of billing for the order is confirmed, 
step 414. Representatively, users may receive an invoice or charge 

15 their order to a credit card, although any known manner of 
effecting payment could be used. Upon generation of an invoice or 
the charging of a credit card account, an "OK-to-go" flag for the 
relevant report is set, step 416. In the preferred embodimejit, the 
review and setting of the "OK-to-go" flag are performed manually, 

2 0 Once the reports have been generated, the report delivery 

device 44 is executed. This program scans each report or set of 
reports in the "out box" or the "print box", as appropriate, step 
418. If the report in the "out box" or the "print box" has the 
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"ready- to-go" flag set, the report delivery device checks to see if 
the "OK-to-go" flag has been set. If the "OK-to-go" flag has been 
set, step 420, the associated report material is emailed to the 
user with a standard, customized cover letter, step 422. Reports 
5 placed in the "print box" are sent to the user by regular mail or 
other hard copy delivery means. Once the reports have been sent, 
the flags are reset and the sent reports are moved from the "out 
box" , or the "print box" , to the "sent box" , step 424 . In this way 
a copy of every report is retained for future reference. 

10 A variety of reports may be generated using the datamart 

system. The reports typically include a combination of text pages 
showing metrics and comparative results, as well as graphic pages 
showing comparative positioning. Representatively, the report 
options that may be generated include, among others, executive 

15 summary, peer group comparison, peer group ranking, peer group 
performance matrix, performance gap analysis, financial gap 
analysis, and gap versus solution optimizer index reports. 

The executive summary report is an overview report that 
compares the requesting user with all database participants. A 

20 representative executive summary report is set forth in Figure 8. 

The peer group comparison report compares the performance 
of the requesting user with that of a peer group on a metric by 
metric basis. The comparison is drawn not only with the peer group 



i 1 

as a whole, but also with the best of the peer group (which may be 
defined as the top 10%) and with all participants in the database. 
Further detail may include an indicator of survey response (number 
of respondents, percentage of response, etc) , as well as median, 
5 average, standard deviation and highest value scores for each of 
the peer group, best-in-peer group and all participants categories. 
The user requesting a peer group comparison report may define 
particular peer group identification parameters to establish a 
customized peer group profile. A sample sheet illustrating a 
10 representative peer group comparison report is presented in Figure 
9. 

The peer group ranking report provides the requesting 
user with a mult i -metric summary identifying how individual metrics 
compare with those of an individualized peer group. The report 

15 includes a percentile rank as well as an overall numeric ranking. 
A sample sheet from a representative peer group ranking report is 
set forth in Figure 10. 

The peer group performance matrix report graphically 
compares the requesting user to a selected group of participants 

20 having similar characteristics. Through a 2-by-2 matrix, this 
report presents efficiency versus effectiveness in a visual 
"balance score card" of performance. Of special importance is the 
fact that the requesting user's business is plotted on the same 

- 27 - 
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matrix with the peer group, allowing the user to immediately assess 
comparative performance. A representative peer group performance 
matrix report is presented in Figure 11. 

The performance gap analysis report allows analysis of 
5 the performance gap between the. requesting user and a peer group by 
identifying comparative performance and then calculating a "value 
creating gap" or a "value destroying gap", as applicable. Looking 
at the sample performance gap analysis report shown in Figure 12, 
for example, in the event the requesting user ranks above the peer 

10 group in a specific metric, e.g., average time in queue, for that 
metric the user will see a percentage in the column entitled 'Value 
creating gap" . This percentage value shows the degree to which the 
user's performance is superior to that of the peer group. The same 
applies to values placed in the column entitled "value destroying 

15 gap", e.g., average after call work time. 

The financial gap analysis report allows analysis of the 
estimated economic cost gap between the requesting user and a peer 
group by identifying comparative performance and then calculating 
an absolute gap and a relative gap. The absolute gap represents 

2 0 the numeric difference between the score of the requesting user and 
that of the peer group, while the relative gap takes into account 
the percentage difference conveyed by such gap. A sample financial 
gap analysis report is shown in Figure 13. 



The financial gap analysis report also includes a 
conversion of the performance gap to a monetary value, the "gap 
value" . This conversion allows the requesting user to prioritize 
those areas needing improvement on the basis of the "bottom line" . 
5 Word formulas used to calculate the value of performance gaps are 
summarized in Tables I and II. 

The gap versus solution optimizer report is a 
sophisticated report that takes each metric for which the user has 
a negative gap relative to the peer group and prints out a summary 

10 of potential solutions available on the market for narrowing or 
eliminating that gap. An approximate cost per agent seat and 
implementation time for each solution is also provided. 
Furthermore, the gap versus solution optimizer report assigns a 
risk factor to each solution, estimates the impact of the solution 

15 on the performance gap, and calculates an expected return on 
investment (ROI) from each solution. Finally, a weighted Optimal 
Decision Index is calculated which takes all of the factors into 
consideration and yields a listing ranked in descending order. A 
sample gap versus solution optimizer report is provided in Figure 

20 14 . 
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In a preferred embodiment, the Optimal Decision Index is 
determined using four measures of specific solutions, namely cost 
to implement, time to implement, risk to implement, and ROI to 
implement. A weighted average is taken of these four measures to 
determine which solution is optimal. 

Through use of the present invention, decision makers are 
able to better understand the performance of their company as a 
whole. By benchmarking specific company processes or functions, on 
an articulated basis, businesses are able to improve their 
performance, which is critical in today's highly competitive 
market. The targeted benchmarking provided by the present 
invention enhances management decision-making abilities in 
selecting the ''best" improvement initiatives. 

The present invention further includes the capability to 
manipulate and mine data contained in the database in real time so 
as to generated tailored informational reports. For example, a 
requesting user could request a report reflecting average 
compensation for call center operators in the state of Arizona. By 
specifying particular data categories or indices, e.g., operator 
compensation, and desired qualifying limitations, e.g., geographic 
area, requesting users are able to obtain a wide range of 
customized reports specifically directed to their own informational 
needs. The data in the database can be manipulated according to 



any indexed data parameter or limitation. 

The foregoing descriptions and drawings should be 
considered as illustrative only of the principles of the invention. 
The invention may be configured in a variety of formats and is not 
5 limited by the on-line presentation of the preferred embodiment. 
Numerous applications of the present invention will readily occur 
to those skilled in the art. Therefore, it is not desired to limit 
the invention to the specific examples disclosed or the exact 
configuration and operation shown and described. Rather, all 
10 suitable modifications and equivalents may be resorted to, falling 
within the scope of the invention. 
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